Grammar Based Directed Testing of Machine Learning Systems

The massive progress of machine learning has seen its application over a variety domains in the past decade. But how do we develop systematic, scalable and modular strategy to validate machine-learning systems? We present, best our knowledge, first approach, which provides systematic test framework for systems that accepts grammar-based inputs. Our Ogma approach automatically discovers erroneous behaviours classifiers leverages these improve respective models. inherent robustness properties present any well trained model direct generation thus, implementing methodology. To evaluate have tested it on three real world natural language processing (NLP) classifiers. found thousands systems. also compare with random observe is more effective than such by up 489 percent.